This paper introduces a model-based reinforcement learning (MBRL) framework that incorporates the underlying decision problem in learning the transition model of the environment. This is in contrast with conventional approaches to MBRL that learn the model of the environment, for example by finding the maximum likelihood estimate, without taking into account the decision problem. Value-Aware Model Learning (VAML) framework argues that this might not be a good idea, especially if the true model of the environment does not belong to the model class from which we are estimating the model. The original VAML framework, however, may result in an optimization problem that is difficult to solve. This paper introduces a new MBRL class of algorithms, called Iterative VAML, that benefits from the structure of how the planning is performed (i.e., through approximate value iteration) to devise a simpler optimization problem. The paper theoretically analyzes Iterative VAML and provides finite sample error upper bound guarantee for it.

Computing an approximately optimal policy with high probability in this case is known as PAC RL with a generative model.

In this paper, we develop efficient parameterized algorithms for regimes between these two extremes.

In a standard view of the reinforcement learning problem, an agent's goal is to efficiently identify a policy that maximizes long-term reward.

These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization.

A.4 ProofofLemma4.5 The proof of Lemma 4.5 depends on another lemma, which will also be useful in the unknown hypothesis class setting. Then there exists somec for whichhc(x) / {0,1}, which is impossible. If qH(x) = c, we don't need to prove anything. Let h H denote the decision maker's decision rule. From Corollary 4.6, we know that h(x)=1 {qH(x) c}aslongasqH(x)6=c.

These instances range from active learning over multi-armed bandits to self-training. We show that all these algorithms not only learn parameters from data but also vice versa: They iteratively alter training data in a way that depends on the current model fit. We introduce reciprocal learning as a generalization of these algorithms using the language of decision theory. This allows us to study under what conditions they converge.